View virtualization and transformations for mobile applications

ABSTRACT

Application mobilization has assumed importance given has assumed importance given the proliferation of smartphones amongst the enterprise workforce, and other users. Existing techniques to mobilize applications incur high overheads in terms of time, cost, and effort. Remote computing is a promising basis for application mobilization that has the advantages of zero code rewriting, full functionality applications, and seamless IT manageability. However, it has the major disadvantage of presenting a cumbersome PC application view on the smartphone. In this context the present invention, referred to as *Mobile, is described to perform dynamic transformation of application views for smartphones. Significant performance gains may be achieved by using this system and methodology.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This application claims priority to U.S. Provisional Application No.61/547,329, filed Oct. 14, 2011 and is a continuation of U.S. patentapplication Ser. No. 13/651,593, filed Oct. 15, 2012, both of which areherein incorporated by reference in their entirety.

BACKGROUND

Enterprise applications represent an important class of softwareapplications ranging in functionality from supply chain management andcustomer relationship management to human resources and payrollmanagement. While such applications are traditionally offered on a PCplatform, recent studies have shown that knowledge workers typicallyspend only one-third of their time physically at their desk, but arevery likely to possess smartphones. Such trends have motivatedenterprises to strategically invest in application mobilization, whereapplications hitherto accessible only on PC platforms are madeaccessible from smartphones as well.

Application mobilization is done today using one of three differentstrategies: (i) homegrown solutions where enterprises directly investresources in developing custom mobilized applications; (ii) third partysolutions where application vendors such as SAP, Oracle, and Microsoftprovide mobilization platforms that can in-turn be used by enterprisesto mobilize applications with appropriate configurations; and (iii)cross platform solutions like web applications that are compatible withmultiple platforms including smartphones. The above methodologies tomobilization however come with their respective limitations as discussednext.

The typical goals of an enterprise for application mobilization are thefollowing: (i) low cost/time to mobilization, (ii) no majorfunctionality loss, (iii) user friendly, and (iv) minimal changes to IT.Unfortunately, homegrown solutions incur considerable time/costoverheads and require major changes to enterprise IT to manage a newclass of applications; third party solutions incur more reasonable, butsubstantial, overheads to configure, typically have much reducedfunctionality if they come pre-designed, and require major changes toenterprise IT processes; and cross platform solutions are typicallyquite cumbersome to use as the interface is originally designed for thePC platform and can support only limited functionalities compared tonatively developed applications.

SUMMARY

It is to be understood that both the following general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive, as claimed.

The present invention may make use of remote computing as the basis formobilization. Mobilizing an application may involve running a PC versionof the application on a physical or virtual PC within the enterprisenetwork and providing a remote view into that application on thesmartphone. Remote computing offers several attractive benefitsincluding zero code mobilization—as no application porting is required;full functionality applications—the entire PC application functionalityis now available for use; and seamless manageability—the backendinfrastructure can be managed with existing IT processes. The use ofremote computing from mobile platforms is not new—Microsoft RemoteDesktop Mobile, VMware View Mobile, and Citrix Receiver Mobile areexamples. However, relying on remote computing as a primary means to useapplications raises a major challenge: remote computing solutionstypically assume homogeneous platforms on either end of the remotesession, which in turn means that application views, originallydeveloped for the PC, will be presented as-is on the smartphone makingthem cumbersome, if not impossible, to use. The present inventionovercomes this challenge.

The present invention comprises a rapid mobilization solution, which wecall *Mobile. The term *Mobile is simply used as a shorthand for thesystem and methodology of the present invention. *Mobile may be builtatop a traditional remote computing solution, or it may be builtindependent of a remote computing solution. However, *Mobile maydynamically transform the application views in a manner that make themappropriate for the smartphone. The *Mobile design, in one embodiment,may comprise two components: (i) virtual view—that virtualizesapplication views independent of the UI framework used into a set of UIelements and attributes that can be easily manipulated; and (ii)transformation services—that can be programmed onto the *Mobile platformto operate on the virtual view during run-time in order to realize asmartphone friendly view. User-studies based on a sample prototypeimplementation show that *Mobile can reduce time-to-task by up to 66%,reduce actions-per-task by up to 85%, and reduce traffic consumption byup to 78%.

In one embodiment of the present invention, mobilization of typicalenterprise applications is performed. Both standalone and client-side ofclient-server windows-based applications may be mobilized, but thedesign principles themselves can be extended to other OSes and webplatforms as well. While the design is independent of the smartphoneplatform (Apple iOS, Android, etc.), in one sample prototypeimplementation, the Android OS for smartphones may be used.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments and together with thedescription, serve to explain the principles of the methods and systems:

FIG. 1 depicts a remote computing architecture.

FIGS. 2a, 2b, and 2c depict performance graphs of remote computerenvironments.

FIG. 3 is a block diagram of the present invention, in one embodiment.

FIGS. 4a and 4b illustrate different ways of adapting view of amobilized

FIGS. 5a, 5b, 5c, and 5d illustrate screenshots of the frontend of thepresent invention, in one embodiment.

FIGS. 6a, 6b, 6c , and 7 are graphs showing the performance of thepresent invention, in various embodiments.

DETAILED DESCRIPTION

The subject disclosure may be understood more readily by reference tothe following detailed description of exemplary embodiments of thesubject disclosure and to the Figures and their previous and followingdescription.

Before the present systems, devices, and methods are disclosed anddescribed, it is to be understood that the terminology used herein isfor the purpose of describing particular embodiments only and is notintended to be limiting.

As used in the specification and the appended claims, the singular forms“a”, “an”, and “the” include plural referents unless the context clearlydictates otherwise.

Ranges may be expressed herein as from “about” one particular value,and/or to “about” another particular value. When such a range isexpressed, another embodiment includes from the one particular valueand/or to the other particular value. Similarly, when values areexpressed as approximations, by use of the antecedent “about”, it willbe understood that the particular value forms another embodiment. Itwill be further understood that the endpoints of each of the ranges aresignificant both in relation to the other endpoint, and independently ofthe other endpoint.

In the subject specification and in the claims which follow, referencemay be made to a number of terms which shall be defined to have thefollowing meanings: “Optional” or “optionally” means that thesubsequently described event or circumstance may or may not occur, andthat the description includes instances where said event or circumstanceoccurs and instances where it does not.

As employed in this specification and annexed drawings, the terms“unit”, “component,” “interface”, “system”, “platform”, and the like areintended to include a computer-related entity or an entity related to anoperational apparatus with one or more specific functionalities, whereinthe computer-related entity or the entity related to the operationalapparatus can be either hardware, a combination of hardware andsoftware, software, or software in execution. One or more of suchentities are also referred to as “functional elements.” As an example, aunit may be, but is not limited to being, a process running on aprocessor, a processor, an object, an executable computer program, athread of execution, a program, a memory (e.g., a hard disc drive),and/or a computer. As another example, a unit can be an apparatus withspecific functionality provided by mechanical parts operated by electricor electronic circuitry which is operated by a software or a firmwareapplication executed by a processor, wherein the processor can beinternal or external to the apparatus and executes at least a part ofthe software or firmware application. In addition or in the alternative,a unit can provide specific functionality based on physical structure orspecific arrangement of hardware elements. As yet another example, aunit can be an apparatus that provides specific functionality throughelectronic functional elements without mechanical parts, the electronicfunctional elements can include a processor therein to execute softwareor firmware that provides at least in part the functionality of theelectronic functional elements. An illustration of such apparatus can becontrol circuitry, such as a programmable logic controller. Theforegoing example and related illustrations are but a few examples andare not intended to be limiting. Moreover, while such illustrations arepresented for a unit, the foregoing examples also apply to a component,a system, a platform, and the like. It is noted that in certainembodiments, or in connection with certain aspects or features thereof,the terms “unit”, “component”, “system”, “interface”, “platform” can beutilized interchangeably.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other additives, components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Reference will now be made in detail to the various embodiments,aspects, and features of the subject disclosure, examples of which areillustrated in the accompanying drawings. Wherever possible, the samereference numbers are used throughout the drawings to refer to the sameor like parts.

Since remote computing is used as a building block in *Mobile, a shortprimer on the technology and how it may be used for mobilization ispresented. FIG. 1 illustrates remote computing from a smartphone 101, inone embodiment. A remote computing server 104 runs on the backend PC102. A remote computing client 103 runs on the smartphone 101, receivesupdates to desktop views from the server 104, and renders them. Theclient 103 sends back any user-input, and the server 104 executes theinput on the PC 102. Virtual Network Computing (VNC) and Remote DesktopProtocol (RDP) are both examples of remote computing solutions. In VNCthe server 104 sends raw pixel information, while in RDP the server 104sends graphical primitives and commands.

Remote computing can be used as a mobilization strategy by simplyrunning the application that requires mobilization on a PC backend 104,and providing a view of that application's bounding box to thesmartphone 101. Attaching the backend PC 102 to the appropriate filesystem can provide access to the user's data stores. VNC may be used asthe underlying platform for the *Mobile system due to the readyavailability of open-source VNC clients. However, the design presentedis agnostic to the specific remote computing solution 104.

There are various advantages to using remote computing for mobilization,including:

Zero software porting: Perhaps the most important advantage ofmobilization using remote computing is that there is no porting of theapplication software required. The application still runs only on a PC,and merely a view is furnished to the smartphone.

Easy IT manageability: A by-product benefit of continuing to use PCapplications even for smartphone users is that the management of themobilization infrastructure can be done using existing IT processes,such as those for software updates.

Familiar interface: Since users have previously relied on the originalPC application 105, the interface furnished through remote computingwill continue to be familiar, and any functionality accomplishable usingthe PC could be performed from the smartphone.

One drawback with using remote computing for mobilization is that theapplication view from the PC is presented as-is on the smartphonewithout any transformations save for resolution scaling. The PC view isquite cumbersome to use on the smartphone, and several reasonscontribute to the unwieldiness of the interface: (i) The bounding box ofthe application on the PC is typically much larger than the screen realestate on the smartphone. This raises a pan/zoom trade-off for the user.In a zoomed out mode, panning to reach different sections of the view isreduced, but the UI elements are too small to read or manipulate easily.Thus, once the user reaches the section of interest, a zooming in isalmost always required. In a zoomed in mode, the user has bettervisibility of the UI elements, but the burden to pan increasesconsiderably; (ii) Interfaces on the smartphone are typically layereddue to the limited screen size. This requires more navigation by theuser. For example, the first level keyboard exposed on smartphones isnot a full keyboard, and requires additional clicks to reach specialkeys; (iii) Independent of the above issues that increase user effort,performing the same number of actions on the smartphone as on the PC isalso subjectively burdensome to the user due to the constrainedenvironment.

To study the above problems quantitatively, user-studies of threeapplications—Microsoft Project, Intuit Quickbooks, and MicrosoftWord—were performed, mobilized using baseline remote computing. Theobjective metric of total number of actions (such as mouseclicks/keyboard entries) taken by each of the 10 participating users tocomplete pre-defined tasks were measured. A set of six commonly usedtasks of varying difficulty levels for each of the three applicationswere used.

FIG. 2a shows the performance when the tasks are performed on thesmartphone and PC respectively (with 90% confidence intervals). Theaverage number of actions required on the smartphone is 3.1× to 4.2× ofthat required for the PC. Similarly FIG. 2b shows the average timerequired to perform the same tasks. Such inflation in user burden candirectly be attributed to the reasons identified earlier.

Another drawback of remote computing is the back and forth exchange ofdata between the server and client that could impose data usage burdenson the wireless link. The traffic consumed by the baseline remotecomputing solution for the three PC applications averaged over the sixtasks per application was studied. The average traffic consumption forthe three applications is presented in FIG. 2c and was 50.33 Kbps duringthe execution of a task. While not prohibitive, the performance would bean issue in cellular data networks where the available per user capacitycan dip below the required data rates. Furthermore, with increasingtrends toward usage based billing in cellular data networks, reducingthe required data rate will also ease cost burdens.

The present invention therefore overcomes these and other limitations ofprior remote computing solutions for mobilized applications.

The overall design of the *Mobile system and methodology will now bedescribed. Unlike traditional remote computing that provides the view toa PC application as it is, *Mobile transforms the application view intoa more mobile-friendly one for the smartphone 101. The key idea is toexpose the application view as a set of UI elements that are easy tomanipulate, and to build transformation services on the virtual view.Two attributes may be used to provide such a virtual view to any PCapplication—(i) most, if not all, PC applications use underlying UIframeworks for the creation of UI elements; and (ii) there arestandardized accessibility approaches to tap into the UI frameworks.However, there may be multiple UI frameworks, and applications could usemultiple frameworks simultaneously. Thus, there is a need forvirtualizing application view into a uniform representation to make theimplementation of transformation services easy. The *Mobile platformarchitecture, as shown in FIG. 3, includes the virtual view 301 as thecore construct and the three high-level components around it—(i) theview abstractor 302 that creates the virtual view for each PCapplication, (ii) the run-time 303 that delivers and renders the virtualview to the smartphone frontend 350, and (iii) the virtual view API 304that makes *Mobile a programmable platform for developing transformationservices, which will be explained further below.

A virtual view 301 is an abstract representation of the user interfaceof a PC application. It may be a tree structure where each nodecorresponds to a UI element in the application, and each link representsthe relationship between a container UI element and another UI elementin the container. Such a tree-like structure naturally exists in any PCapplication created with a standard UI framework. Specifically, the rootnode of a virtual view represents the main window of a PC application.The window typically contains a menu bar, a tool bar, a status bar, andother UI elements. The children nodes further contain children nodes.For example, the menu bar contains several menu items, which may furthercontain sub-menu items.

For each node in a virtual view 301, we may define a set of attributesthat can be used to describe a UI element created with a standard UIframework Table 1 lists seven common attributes that may be extractedfrom the application view by the UI abstractor—(i) id is a uniqueidentifier for the UI element; (ii) the type of a UI element can bebutton, menu item, text field, etc. (iii) location describes thecoordinate of the UI element in the application view; (iv) size containsthe width and height of the UI element; (v) state means informationcontained inside the UI element, such as the text in a text field or thechecked state of a check box; and (vi) parent and children describe thehierarchy relationship in the tree structure of the virtual view. Besidethose extracted from the application view, the virtual view 301 may addtwo attributes to the node that will be used by the run-time 303. Thestatus of a node indicates the UI element to be shown or hidden by therun-time, and the template provides style descriptions in rendering theUI element.

TABLE 1 Widget toolkit NET SAP Web Accessibility framework UI AutomationSAP GUI Scripting JavaScript and DOM Attributes Id (Name,LocalizedControlType, (Name, Type, Id) (Name, TagName, Id) AutomationId)Type LocalizedControlType Type TagName Location (Left, Top) (Left, Top)(OffsetLeft, OffsetTop, OffsetParent) Size (Right-Left, Bottom-Top)(Width, Height) (OffsetWidth, OffsetHeight) State Value and State Valueand State Attributes Parent Ancestors, head Parent ParentNode ChildrenChildren Children ChildrenNode Status N/A: virtual view only TemplateN/A: virtual view only Functions Read Read from an attribute Write Writefrom an attribute Invoke Invoke Execute Click Events OnOpenMenuOpenedEvent ContextMenuEvent DOMNodeInserted WindowOpenedEventChangeEvent DOMNodeInsertedIntoDocument OnClose MenuClosedEventDestroyMenuEvent DOMNodeRemoved WindowClosedEvent DestroyEventDOMNodeRemovedFromDocument OnActivity FromPoint, FocusedElementFindByPosition, GUIFocused OnClick, OnDblclick, OnKeyUp

Besides the abovementioned static information, a virtual view 301 mayalso carry dynamic information in terms of the events that happen in thePC application 105. As shown in Table I, three types of common eventsmay be captured:—(i) an OnOpen event occurs when a view, such as apop-up menu or a dialog, is opened in the PC application 105; (ii) anOnClose event occurs when a view is closed in the PC application 105;and (iii) an OnActivity event occurs whenever the user performsactivity, such as a mouse click or a keystroke, on a UI element.

The UI abstractor 302 establishes a virtual view for each PC application105 by extracting the information from the underlying UI framework in agenetic, robust, and efficient manner. Most PC applications 105construct their application view via a framework called widget toolkitthat provides reusable modules for creating UI elements. The informationof the UI elements created by a widget toolkit is typically accessibleusing a corresponding accessibility framework, which is used inassistive technology (ex. a screen reader) and automated UI testing insoftware development. For example, UI Automation provides viewinformation of applications developed with Microsoft .NET framework.JavaScript provides access to the Document Object Model (DOM), whichdescribes the UI elements in any web application. Enterprise applicationplatforms also have their own accessibility frameworks (ex: SAP'saccessibility framework is called SAP GUI Scripting. Similarly, otherpopular widget toolkits such as GTK+, Cocoa, Qt, wxWidgets, Java, andFlash all have corresponding accessibility frameworks.

The UI abstractor 302 may create a uniform representation of virtualviews 301 for the mobilized applications by leveraging the accessibilityframeworks. Although each accessibility framework has its own API, ittypically provides UI information that may match with the genericrepresentation of the virtual views 301. Table 1 shows how theattributes and events may be matched with those used in UI Automation,SAP GUI scripting, and JavaScript.

The UI abstractor 302 provides several advantages in virtualizingdifferent accessibility frameworks into a common representation: (i) asingle implementation of any view transformation service based on theuniform interface can work for multiple applications; (ii) it enablesfederated applications by allowing the view transformation services towork across multiple applications, examples of which are provided laterwith respect to the aggregation and mashup transformation service; and(iii) it creates a single view for an application that uses multiplewidget toolkits. Examples include using a web application in a browser(.NET+web), applications with embedded web pages (SAP+web), webapplications with embedded objects (web+Java applet and web+Flash).

Two core services may be used in providing a mobile-friendly view atrun-time 303: delivery of the virtual view to the frontend 350, andrendering of the virtual view in the frontend 350. First, the interfacetransformation service renders the virtual view 301 as a more amenableand appropriate interface for the smartphone. It transforms each UIelement in a virtual view 301 into a native UI element if acorresponding widget module is available in the smartphone platform;otherwise the element is called workspace and is rendered with theunderlying remote computing. For example, mouse clickable tabs could betransformed into touchable buttons, and a pull down menu could betransformed into a spinner wheel. Rendering with native elementsprovides advantages including instant response in terms of theclient-side UI interaction and a consistent look-and-feel as asmartphone app. Second, the traffic suppression service efficientlydelivers the virtual view 301 to the frontend 350. While remotecomputing is needed for the workspace, the virtual view 301 is leveragedto intelligently suppressing and compressing the traffic consumption ofunderlying remote computing with three techniques: reducing the scope ofscreen update to the workspace, suppressing unnecessary requests for thetransformed views, and leverage the long-term redundancy across multiplescreen views with multi-frame encoding/decoding.

The virtual view provides a simple API 304 to allow easy implementationof view transformation services that further improves the performance ofthe mobilized applications. As shown in Table 1, the API 304 may providethree straightforward functions: the Read and Write functions allow aservice to access the value of an attribute of a UI element, and theInvoke function allows a service to perform activity on a UI element,such as clicking a button. Besides, the API 304 allows any service toregister to any event, and the service will be notified using a callbackmechanism when an event occurs in the PC application.

Several feature-rich add-on view transformation services can be built ontop of the virtual view API 304. FIGS. 4a and 4b illustrate fivedifferent ways of adapting view of a mobilized application on asmartphone. The five services provide complementary functionality andcan be used in tandem. While five add-on services are defined, these areby no means a comprehensive set. The five add-on services demonstratethe power of the virtual view API 304 described in the previous sectionand also show how the basic view transformation can be enhanced. Forease of deployment, all services may be hosted at the backend 351 andthus can be enabled for users without the need to make any change to thefrontend 350 that has been installed in their smartphones.

The Aggregation service 401 allows a user to aggregate redundantactivity while using the mobilized application. The service allows auser to aggregate actions for routine tasks and record them in to asingle macro. A macro can then be accessed from the *Mobile frontend 350to automate and also speed up the corresponding task execution.

If there are routine tasks that the user wants to perform from thesmartphone, the service allows users to record a sequence of actions ondifferent UI elements as a macro. The recording can be done on a PC thatis much easier to manipulate, but it can also be done on a smartphone.While using the mobilized version of the PC application, macros recordedby the user are presented in the transformed view, and the user canreplay the macro to quickly accomplish the same task from thesmartphone.

The macro recording may be realized by listening to the OnActivityevents and reading the id attributes of the associated UI elements. Thelist of macros recorded for a mobilized application may be presented asmenu items in the menu bar. Playback of a macro may be realized byinvoking the constituent UI elements with the recorded id attributes.The Aggregate service 401 can also be extended further to supportparameters or custom values for certain actions during playback of amacro.

The macro presentation may be realized by writing new UI elements aschildren of a container node, such as menu items in the menu bar. TheAggregation service 401 can also support parameterization that allows auser to provide a different value for a parameter UI element in aroutine task, such as the file name when saving a file. A parameter UIelement can be identified by reading the type attribute, and the settingof the parameter value in runtime can be done by writing the valueattribute of the UI element.

The Reduction service 402 provides a simplified view of the mobilizedapplication by showing only a subset of the UI elements that the userwants to access from the smartphone. This helps to reduce a possibleclutter of UI elements by eliminating some unused UI elements from theview. The reduction can be performed in one of the following ways: (a)manually choosing the useful UI elements or (b) automatically choosingonly the frequently used subset of UI elements.

In one embodiment, two approaches may be used in deciding which UIelements to show: (a) the service allows the user to manually choose tohide certain UI elements from the transformed view; and (b) the serviceautomatically choose the frequently used subset of UI elements to beshown in the smartphone.

The concept of reduction is to provide a simple application view in themobilized application by showing only a subset of UI elements that theuser wants to access from the smartphone.

The Reduction service 402 may be realized as follows. First, itregisters to the OnOpen events. It writes the status attribute of the UIelements in a virtual view as shown or hidden to configure the subset ofUI elements to be shown from the smartphone. It also uses the API todecide which UI elements to reduce. The service can provide an “editmode” by adding a “show/hide” toggle button adjacent to each UI element.It can also listen to the OnActivity events to analyze user activity forautomated reduction.

The Rearrangement service 403 presents a customized layout for thetransformed view and allows configuration of the placement of thevarious UI elements in the view. This is particularly useful if theusage patterns of the mobilized application is different from the PCapplication and thus the original layout is no longer suitable.

The rearrangement service 403 may be realized by moving the nodes in thetree structure of the virtual view 301 by writing the childrenattribute. Similar to the reduction service 402, rearrangement 403 canbe performed either manually or automatically, and either approach maybe supported by the same functions in the virtual view API 304.

The Customized Transformation service 404 allows users to furthercustomize the transformed view of the application. It allows a user tomodify the look and feel of different UI elements by defining a customtemplate. This custom template may be used by the frontend 350 to renderthe different UI elements during runtime. In an Android environment, thetemplate can be defined as an XML snippet that describes the stylesettings such as size, padding, margin, layout, look-and-feel, etc.

The concept of this service is to allow users to customize thetransformation of the virtual view 301 in the smartphone. This may beperformed by writing the template attribute of certain UI elements. Thetemplate is the metadata used to describe how a native element isrendered in the smartphone platform. In Android, the template may be anXML snippet that describes the characteristics such as size, padding,margin, layout, look-and-feel, etc.

The Mashup service 405 allows the user to federate UI elements from twoor more different applications and show them as a single view on thesmartphone. The federated application view allows the user to performactivity across multiple PC applications and/or get simultaneous viewinto them in a dash board-like smartphone app. The service may berealized by reading the virtual view of one application and insertingits UI elements as children of certain UI elements of anotherapplication.

The Overflow service 406 intelligently splits a virtual view intomultiple views to suit the smartphone environment. Because of thedifference in form factor, the user interface of a PC applicationtypically has more elements than that of a mobile app. When a virtualview 301 contains more elements than what can be fit into a smartphonescreen, squeezing all elements into the small screen would result in adense and unusable user interface. Overflow service intelligentlyorganizes elements in a virtual view 301, presents a suitable number ofelements for the smartphone screen, and hides the remaining elements inan “overflow” view visible on demand.

The Overflow service 406 improves the usability of a dense virtual viewof lots of elements by intelligently splitting a virtual view intomultiple views, including the main view and the overflow view. Only aset of number of elements are presented in the main view to suit thesmall screen of the smartphone, and the remaining elements are movedinto “overflow” views that can be made visible on demand. Similar to thereduction service, the elements in the main view can be manuallyconfigured by the user, or automatically configured by the overflowservice based on the usage frequency of the elements.

In one embodiment, there are two ways that an overflow view can becreated for the remaining elements. The first one is a pop-up dialogthat will be layered on top of the main view when invoked. The secondone is an in-line expansion where the elements are stored in aninvisible container, which is made visible when invoked. Duringrun-time, the service registers to the OnOpen( ) and OnUpdate( ) eventsto reorganize the elements in the virtual view.

The Zoom service 407 allows a user to dynamically adjust the set of UIelements available in the mobile app from the frontend 350 on demand. Itallows the user to define different zoom levels for a virtual view,ranging from simple but feature-limited ones with fewer elements, tofeature-rich but complex ones with more elements. It may provide acontrol knob in the frontend 350 to allow the user to dynamically adjustthe zoom level in run-time depending on what features the user wants toaccess.

The Zoom service 407 allows a user to adjust richness of the frontend350 with several zoom levels. The lowest zoom level provides thesimplest user interface with the most limited set of features, and thehighest zoom level enables all features that the user wants to accessfrom a smartphone. Each element in a virtual view 301 is associated to azoom level. The zoom service registers to OnOpen( ) and OnUpdate( )events and adds a knob to allow the user to adjust the zoom level fromthe frontend 350. When the user switches to a certain zoom level, allelements at a higher zoom level are removed from the virtual view 301,and then the updated virtual view 301 is sent to the frontend 350.

While the design of *Mobile system of the present invention can beapplied to mobilize any PC application to any smartphone platform, inone embodiment Windows and Android may be used as the platforms. The UIabstractor 302 may construct the virtual view 301 with UI Automation,which is the most widely supported accessibility framework in Windows.The two core services in the run-time-interface transformation andtraffic suppression—may be implemented with VNC as the underlying remotecomputing. TightVNC server may be used at the backend 351, andAndroidVNC may be used at the frontend 350. One of the add-on services,aggregation 401, may be implemented on top of the virtual view 301.

While any type of client device may be used, in one embodiment a SamsungGalaxy S smartphone running Android 2.1 withal GHz CPU and 512 MB RAMmay be used. The application proxy may be a Windows Server 2003. The*Mobile system may be built, in one of a variety of embodiments, bymodifying the open-source AndroidVNC at the frontend 350 and TightVNCserver at the backend 351. In one embodiment, three enterpriseapplications—namely Quickbooks Enterprise Solutions, MS Project and MSWord—are mobilized.

In FIGS. 5a and 5b . representative screenshots of the *Mobile frontend350, in one embodiment, are depicted. FIGS. 5a and 5b show the defaultremote computing view and the transformed view of a menu of theQuickbooks application, respectively. How the different menu items havebeen transformed into a spinner interface on the smartphone can be seen.FIG. 5c shows a transformed dialog of MS Project. The transformed dialogshows different VI elements of the Android V I framework, such asbuttons, combo boxes and text labels.

The performance of the interface transformation service were evaluated.Ten volunteers were invited from both academia and industry. Thevolunteers were picked to represent diversity across age-groups (25-40),gender and employer type (university, start-up, and large enterprise).Eight of the volunteers were heavy smartphone users. Each user was askedto perform one representative task for each of the three applicationsand using all three interfaces (PC, baseline mobilization based on VNC,and *Mobile).

FIG. 6a shows the average time taken in accomplishing a task of theapplications from the smartphone (with 90% confidence interval). Thetime taken on a PC is also shown in the figure as the benchmarkcomparison. Since the user interface in the PC applications is alreadyoptimized, the time-to-task we can achieve in a mobilized app can onlyequal to or greater than that of PC. As shown in the figure, the *Mobilesystem of the present invention provides up to 66% of the achievableperformance enhancement from smartphones. FIG. 6b shows the average taskeffort, which is defined as the number of actions required inaccomplishing a task. *Mobile achieves up to 85% of the achievableperformance enhancement.

With respect to data traffic consumption, the traffic consumption wasmeasured when the users used the baseline and *Mobile from thesmartphone in the experiments. As shown in FIG. 6c , the trafficsuppression service in *Mobile is able to reduce the traffic size bymore than half in Microsoft Project and Microsoft Word, and about onethird of traffic is reduced in Intuit QuickBooks.

The reason for the better reduction ratio in Project and Word may beattributed by the higher redundancy existing in the usage of the twoapplications. Common operations in the two application cause switchesbetween different views and this results in frequent redundant updates.For example, users switch between different Ribbon tool bars when usingboth applications. Here the multi-screen buffering algorithm helps inreducing the traffic. While menus in QuickBooks also cause redundantupdates, those updates have a smaller size.

FIG. 5d shows a list of all available macros that are created using theaggregation service 401. Since the performance of aggregation may bemore dependent on the nature of the aggregated operations, a moreextensive experiment was conducted by inviting 22 volunteer users toperform 54 tasks using 9 PC applications. FIG. 7 shows the performancein terms of the task effort and time-to-task. Aggregation is able toreduce task effort by 83% and time-to-task by 58%.

The various embodiments of the present systems and methods can beoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that can besuitable for use with the systems and methods comprise, but are notlimited to, personal computers, server computers, laptop devices orhandheld devices, and multiprocessor systems. Additional examplescomprise wearable devices, mobile devices, set top boxes, programmableconsumer electronics, network PCs, minicomputers, mainframe computers,distributed computing environments that comprise any of the abovesystems or devices, and the like.

The processing effected in the disclosed systems and methods can beperformed by software components. The disclosed systems and methods canbe described in the general context of computer-executable instructions,such as program modules, being executed by one or more computers orother computing devices. Generally, program modules comprise computercode, routines, programs, objects, components, data structures, etc.,that perform particular tasks or implement particular abstract datatypes. The disclosed methods also can be practiced in grid-based anddistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules can be located inboth local and remote computer storage media including memory storagedevices.

While the systems, devices, apparatuses, protocols, processes, andmethods have been described in connection with exemplary embodiments andspecific illustrations, it is not intended that the scope be limited tothe particular embodiments set forth, as the embodiments herein areintended in all respects to be illustrative rather than restrictive.Unless otherwise expressly stated, it is in no way intended that anyprotocol, procedure, process, or method set forth herein be construed asrequiting that its acts or steps be performed in a specific order.Accordingly, in the subject specification, where description of aprocess or method does not actually recite an order to be followed byits acts or steps or it is not otherwise specifically recited in theclaims or descriptions of the subject disclosure that the steps are tobe limited to a specific order, it is no way intended that an order beinferred, in any respect. This holds for any possible non-express basisfor interpretation, including: matters of logic with respect toarrangement of steps or operational flow; plain meaning derived fromgrammatical organization or punctuation; the number or type ofembodiments described in the specification or annexed drawings, or thelike.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the subject disclosurewithout departing from the scope or spirit of the subject disclosure.Other embodiments of the subject disclosure will be apparent to thoseskilled in the art from consideration of the specification and practiceof the subject disclosure as disclosed herein. It is intended that thespecification and examples be considered as non-limiting illustrationsonly, with a true scope and spirit of the subject disclosure beingindicated by the following claims.

1. A method for transforming an application user interface to a mobileuser interface on a mobile device, comprising: a. extracting applicationuser interface elements from the application user interface; b.virtualizing the extracted application user interface elements togenerate a virtual view; c. applying a sequence of transformations onthe virtual view; d. transmitting to the mobile device the transformedvirtual view; and e. rendering the transformed virtual view on themobile device.
 2. A method for transforming an application userinterface to a mobile user interface on a mobile device, comprising: a.extracting application user interface elements from the application userinterface; b. virtualizing the extracted application user interfaceelements to generate a virtual view; c. applying a sequence oftransformations on the virtual view; and d. transmitting to the mobiledevice the transformed virtual view, whereby the transformed virtualview is rendered on the mobile device.
 3. A system for transforming anapplication user interface to a mobile user interface on a mobiledevice, comprising: a. means for extracting application user interfaceelements from the application user interface; b. means for virtualizingthe extracted application user interface elements to generate a virtualview; c. means for applying a sequence of transformations on the virtualview; d. means for transmitting to the mobile device the transformedvirtual view; and e. means for rendering the transformed virtual view onthe mobile device.